AWS EKS Data Connector (via Codeless Connector Framework)
Browse: 🏠 · Solutions · Connectors · Methods · Tables · Content · Parsers · ASIM Parsers · ASIM Products · 📊
| Attribute | Value |
|---|---|
| Connector ID | AWSEKSDefinition |
| Publisher | Amazon Web Services |
| Used in Solutions | AWS EKS |
| Collection Method | CCF |
| Connector Definition Files | AWSEKS_ConnectorDefinition.json |
| CCF Configuration | AWSEKS_PollingConfig.json |
| CCF Capabilities | AmazonWebServicesS3 |
The AWS EKS data connector provides the capability to ingest audit logs from Amazon Elastic Kubernetes Service into Microsoft Sentinel. This connector focuses on EKS audit logs (JSON format) which contain detailed information about API server requests, authentication decisions, and cluster activities. The connector uses AWS SQS to receive notifications when new audit log files are exported to S3, ensuring real-time security monitoring and compliance tracking for your Kubernetes clusters.
Tables Ingested
This connector ingests data into the following tables:
| Table | Transformations | Ingestion API | Lake-Only |
|---|---|---|---|
AWSEKSLogs_CL |
? | ✓ | ? |
💡 Tip: Tables with Ingestion API support allow data ingestion via the Azure Monitor Data Collector API, which also enables custom transformations during ingestion.
Permissions
Resource Provider Permissions: - Workspace (Workspace): Read and Write permissions are required.
Setup Instructions
⚠️ Note: These instructions were automatically generated from the connector's user interface definition file using AI and may not be fully accurate. Please verify all configuration steps in the Microsoft Sentinel portal.
1. AWS CloudFormation Deployment
Use the provided CloudFormation templates to configure the AWS environment for sending logs from AWS EKS to your Log Analytics Workspace.
Deploy CloudFormation Templates in AWS:
- Navigate to the AWS CloudFormation Stacks.
- Click Create stack and select With new resources.
- Choose Upload a template file, then click Choose file to upload the appropriate CloudFormation template(Template 1 and 2 below) provided.
- Follow the prompts and click Next to complete the stack creation.
- After the stacks are created, navigation to the Outputs section. Run the scripts in step 1 and 2 from the output section, it stream log from eks to sqs.
- In the same outputs section, Note down the Role ARN and SQS Queue URL which are going to be used in connect connector.
- Template 1: OpenID Connect authentication provider deployment:
{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "This Stack creates an Open ID Connect (OIDC) web identity provider and an AWS assumed role.", "Resources": { "OIDCIdentityProvider": { "Type": "AWS::IAM::OIDCProvider", "DeletionPolicy": "Retain", "Properties": { "ClientIdList": [ "api://1462b192-27f7-4cb9-8523-0f4ecb54b47e" ], "ThumbprintList": [ "626d44e704d1ceabe3bf0d53397464ac8080142c" ], "Url": "https://sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d/" } } } } - Template 2: AWS EKS Resources Deployment:
{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "Production EKS audit log collection infrastructure with updated Lambda function that uses the SUCCESSFUL format for Microsoft Sentinel ingestion. Includes the extra 'date' field and simplified RawEvent structure that works with Sentinel.", "Parameters": { "EKSClusterName": { "Type": "String", "Description": "Enter the name of your existing EKS cluster", "MinLength": 1, "ConstraintDescription": "EKS cluster name is required" }, "AwsRoleName": { "Type": "String", "Description": "Enter the ARN name for the role. The name must start with 'OIDC_', otherwise the connector will not function properly.", "AllowedPattern": "OIDC_[-_a-zA-Z0-9]+", "Default": "OIDC_MicrosoftSentinelRoleEKS" }, "BucketName": { "Type": "String", "AllowedPattern": "^[a-z0-9][a-z0-9-.]{1,61}[a-z0-9]$", "Description": "Enter the name of the S3 bucket. Bucket name must be unique within the global namespace and follow the bucket naming rules.", "Default": "microsoft-sentinel-eks-logs" }, "SentinelSQSQueueName": { "Default": "MicrosoftSentinelEKSSqs", "Type": "String", "Description": "Enter the name for the SQS Queue." }, "SentinelWorkspaceId": { "Type": "String", "Description": "Enter the Microsoft Sentinel Workspace ID" } }, "Resources": { "SentinelWebIdentityBasedRole": { "Type": "AWS::IAM::Role", "Properties": { "RoleName": { "Ref": "AwsRoleName" }, "Description": "Role to provide Microsoft Sentinel access to S3 bucket and SQS queue for EKS logs.", "Path": "/", "AssumeRolePolicyDocument": { "Fn::Sub": "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Effect\": \"Allow\",\"Principal\": {\"Federated\": \"arn:aws:iam::${AWS::AccountId}:oidc-provider/sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d/\"},\"Action\": \"sts:AssumeRoleWithWebIdentity\",\"Condition\": {\"StringEquals\": {\"sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d/:aud\": \"api://1462b192-27f7-4cb9-8523-0f4ecb54b47e\",\"sts:RoleSessionName\": \"MicrosoftSentinel_${SentinelWorkspaceId}\"}}}]}" }, "Policies": [ { "PolicyName": "SentinelS3SQSAccessPolicy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": { "Fn::Sub": "arn:aws:s3:::${BucketName}/*" } }, { "Effect": "Allow", "Action": [ "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes", "sqs:GetQueueUrl", "sqs:ChangeMessageVisibility" ], "Resource": { "Fn::GetAtt": [ "SentinelSQSQueue", "Arn" ] } } ] } } ] } }, "SentinelSQSQueue": { "Type": "AWS::SQS::Queue", "DeletionPolicy": "Delete", "Properties": { "QueueName": { "Ref": "SentinelSQSQueueName" }, "Tags": [ { "Key": "Bucket", "Value": { "Ref": "BucketName" } }, { "Key": "EKSCluster", "Value": { "Ref": "EKSClusterName" } } ] } }, "SentinelSQSQueuePolicyForS3": { "Type": "AWS::SQS::QueuePolicy", "Properties": { "PolicyDocument": { "Statement": [ { "Sid": "Allow S3 to send notification messages to SQS queue", "Action": [ "SQS:SendMessage" ], "Effect": "Allow", "Resource": [ { "Fn::GetAtt": [ "SentinelSQSQueue", "Arn" ] } ], "Principal": { "Service": "s3.amazonaws.com" }, "Condition": { "StringEquals": { "aws:SourceAccount": { "Fn::Sub": "${AWS::AccountId}" } }, "ArnLike": { "aws:SourceArn": { "Fn::Sub": "arn:${AWS::Partition}:s3:*:*:${BucketName}" } } } }, { "Sid": "Allow Assumed role to read/delete/change visibility of SQS messages and get queue url.", "Action": [ "SQS:ChangeMessageVisibility", "SQS:DeleteMessage", "SQS:ReceiveMessage", "SQS:GetQueueUrl" ], "Effect": "Allow", "Resource": [ { "Fn::GetAtt": [ "SentinelSQSQueue", "Arn" ] } ], "Principal": { "AWS": [ { "Fn::GetAtt": [ "SentinelWebIdentityBasedRole", "Arn" ] } ] } } ] }, "Queues": [ { "Ref": "SentinelSQSQueue" } ] } }, "S3Bucket": { "Type": "AWS::S3::Bucket", "DependsOn": "SentinelSQSQueuePolicyForS3", "Properties": { "BucketName": { "Ref": "BucketName" }, "OwnershipControls": { "Rules": [ { "ObjectOwnership": "BucketOwnerEnforced" } ] }, "VersioningConfiguration": { "Status": "Enabled" }, "NotificationConfiguration": { "QueueConfigurations": [ { "Queue": { "Fn::GetAtt": [ "SentinelSQSQueue", "Arn" ] }, "Event": "s3:ObjectCreated:*", "Filter": { "S3Key": { "Rules": [ { "Name": "prefix", "Value": { "Fn::Sub": "eks-audit-logs/${EKSClusterName}/" } } ] } } } ] } } }, "S3BucketPolicy": { "Type": "AWS::S3::BucketPolicy", "DeletionPolicy": "Delete", "Properties": { "Bucket": { "Ref": "S3Bucket" }, "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "Allow Arn read access to S3 bucket.", "Effect": "Allow", "Principal": { "AWS": { "Fn::GetAtt": [ "SentinelWebIdentityBasedRole", "Arn" ] } }, "Action": "s3:GetObject", "Resource": { "Fn::Join": [ "", [ { "Fn::GetAtt": [ "S3Bucket", "Arn" ] }, "/*" ] ] } } ] } } }, "FirehoseDeliveryStreamRole": { "Type": "AWS::IAM::Role", "DeletionPolicy": "Delete", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "TrustDataFirehoseService", "Effect": "Allow", "Principal": { "Service": "firehose.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Policies": [ { "PolicyName": "MSSentinelEKSFirehoseDeliveryStreamPolicy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ { "Fn::GetAtt": [ "S3Bucket", "Arn" ] }, { "Fn::Join": [ "", [ { "Fn::GetAtt": [ "S3Bucket", "Arn" ] }, "/*" ] ] } ] }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction" ], "Resource": [ { "Fn::GetAtt": [ "EKSLogTransformFunction", "Arn" ] } ] } ] } } ] } }, "EKSAuditLogsDeliveryStream": { "Type": "AWS::KinesisFirehose::DeliveryStream", "DeletionPolicy": "Delete", "Properties": { "DeliveryStreamType": "DirectPut", "DeliveryStreamEncryptionConfigurationInput": { "KeyType": "AWS_OWNED_CMK" }, "ExtendedS3DestinationConfiguration": { "BucketARN": { "Fn::GetAtt": [ "S3Bucket", "Arn" ] }, "BufferingHints": { "SizeInMBs": 128, "IntervalInSeconds": 300 }, "CompressionFormat": "GZIP", "FileExtension": ".gz", "Prefix": { "Fn::Sub": "eks-audit-logs/${EKSClusterName}/" }, "ProcessingConfiguration": { "Enabled": true, "Processors": [ { "Type": "Lambda", "Parameters": [ { "ParameterName": "LambdaArn", "ParameterValue": { "Fn::GetAtt": [ "EKSLogTransformFunction", "Arn" ] } } ] } ] }, "RoleARN": { "Fn::GetAtt": [ "FirehoseDeliveryStreamRole", "Arn" ] } } } }, "CloudWatchLogsRole": { "Type": "AWS::IAM::Role", "DeletionPolicy": "Delete", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "TrustCloudWatchLogsService", "Effect": "Allow", "Principal": { "Service": { "Fn::Sub": "logs.${AWS::Region}.amazonaws.com" } }, "Action": "sts:AssumeRole" } ] }, "Policies": [ { "PolicyName": "MSSentinelEKSCloudWatchLogsPolicy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "firehose:PutRecord", "firehose:PutRecordBatch" ], "Resource": [ { "Fn::GetAtt": [ "EKSAuditLogsDeliveryStream", "Arn" ] } ] } ] } } ] } }, "EKSLogTransformFunctionRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" ] } }, "EKSLogTransformFunction": { "Type": "AWS::Lambda::Function", "Properties": { "FunctionName": { "Fn::Sub": "${AWS::StackName}-EKSLogTransform" }, "Runtime": "python3.9", "Handler": "index.lambda_handler", "Role": { "Fn::GetAtt": [ "EKSLogTransformFunctionRole", "Arn" ] }, "Timeout": 60, "Code": { "ZipFile": "import json\nimport base64\nimport gzip\nimport logging\nfrom datetime import datetime\n\nlogger = logging.getLogger()\nlogger.setLevel(logging.INFO)\n\ndef lambda_handler(event, context):\n \"\"\"\n Production Lambda function for EKS audit log processing.\n Transforms real EKS audit logs to Microsoft Sentinel AWSEKSLogs_CL format.\n Based on successful dummy function format that works with Sentinel ingestion.\n \"\"\"\n \n logger.info(\"=== PRODUCTION EKS AUDIT LOG TRANSFORMATION ===\")\n \n output = []\n \n for record in event['records']:\n record_id = record['recordId']\n \n try:\n # Decode the incoming data\n compressed_payload = base64.b64decode(record['data'])\n uncompressed_payload = gzip.decompress(compressed_payload)\n \n # Parse CloudWatch log data\n log_data = json.loads(uncompressed_payload)\n \n transformed_records = []\n \n # Process each log event\n for log_event in log_data.get('logEvents', []):\n try:\n # Parse the EKS audit log message\n eks_audit_log = json.loads(log_event['message'])\n \n # Transform to Log Analytics format using successful schema\n transformed_record = transform_eks_to_la_format(eks_audit_log, log_event)\n \n if transformed_record:\n transformed_records.append(transformed_record)\n \n except Exception as e:\n logger.error(f\"Error parsing log event: {str(e)}\")\n continue\n \n # Encode all transformed records\n if transformed_records:\n # Join multiple records with newlines\n output_data = '\\n'.join([json.dumps(record) for record in transformed_records]) + '\\n'\n encoded_data = base64.b64encode(output_data.encode('utf-8')).decode('utf-8')\n \n output_record = {\n 'recordId': record_id,\n 'result': 'Ok',\n 'data': encoded_data\n }\n else:\n # No valid records, return empty but successful\n output_record = {\n 'recordId': record_id,\n 'result': 'Ok', \n 'data': base64.b64encode(b'').decode('utf-8')\n }\n \n output.append(output_record)\n logger.info(f\"Successfully processed {len(transformed_records)} records from recordId {record_id}\")\n \n except Exception as e:\n logger.error(f\"Error processing record {record_id}: {str(e)}\")\n \n # Return processing failed for this record\n output_record = {\n 'recordId': record_id,\n 'result': 'ProcessingFailed'\n }\n output.append(output_record)\n \n logger.info(f\"Processed {len(output)} total records\")\n return {'records': output}\n\ndef transform_eks_to_la_format(eks_audit_log, log_event):\n \"\"\"\n Transform EKS audit log to Microsoft Sentinel AWSEKSLogs_CL format.\n Uses the successful format that works with Sentinel ingestion.\n \"\"\"\n \n try:\n # Extract timestamp - use requestReceivedTimestamp or fallback to CloudWatch timestamp\n timestamp = eks_audit_log.get('requestReceivedTimestamp')\n if not timestamp:\n # Convert CloudWatch timestamp (milliseconds) to ISO format\n cw_timestamp = log_event.get('timestamp', 0)\n timestamp = datetime.utcfromtimestamp(cw_timestamp / 1000).strftime('%Y-%m-%dT%H:%M:%S.%fZ')\n \n # Extract basic fields with safe defaults\n verb = eks_audit_log.get('verb', '')\n stage = eks_audit_log.get('stage', '')\n \n # Extract user information\n user_info = eks_audit_log.get('user', {})\n username = user_info.get('username', '') if isinstance(user_info, dict) else str(user_info) if user_info else ''\n \n # Extract source IPs\n source_ips = eks_audit_log.get('sourceIPs', [])\n if not isinstance(source_ips, list):\n source_ips = [str(source_ips)] if source_ips else []\n \n # Extract object reference\n object_ref = eks_audit_log.get('objectRef', {})\n if isinstance(object_ref, dict):\n # Build ObjectRef in simple format like working file\n namespace = object_ref.get('namespace', '')\n resource = object_ref.get('resource', '') \n name = object_ref.get('name', '')\n \n if namespace and resource and name:\n object_ref_str = f\"{namespace}/{resource}/{name}\"\n elif resource and name:\n object_ref_str = f\"{resource}/{name}\"\n elif name:\n object_ref_str = name\n else:\n object_ref_str = ''\n else:\n object_ref_str = str(object_ref) if object_ref else ''\n \n # Extract response code\n response_status = eks_audit_log.get('responseStatus', {})\n response_code = 0\n if isinstance(response_status, dict):\n response_code = response_status.get('code', 0)\n \n # Extract authorization decision\n annotations = eks_audit_log.get('annotations', {})\n auth_decision = ''\n if isinstance(annotations, dict):\n auth_decision = annotations.get('authorization.k8s.io/decision', '')\n \n # Create simple RawEvent structure (like successful dummy function)\n simple_raw_event = {\n \"verb\": verb,\n \"user\": username,\n \"objectRef\": object_ref_str,\n \"responseCode\": response_code\n }\n \n # Build the complete record with extra date field (like successful format)\n transformed_record = {\n \"date\": timestamp, # Extra field that makes Sentinel ingestion work\n \"TimeGenerated\": timestamp,\n \"AwsAccountId\": eks_audit_log.get('annotations', {}).get('aws.amazon.com/account-id', '') if isinstance(eks_audit_log.get('annotations'), dict) else '',\n \"Region\": eks_audit_log.get('annotations', {}).get('aws.amazon.com/region', '') if isinstance(eks_audit_log.get('annotations'), dict) else '',\n \"ClusterName\": eks_audit_log.get('annotations', {}).get('aws.amazon.com/cluster-name', '') if isinstance(eks_audit_log.get('annotations'), dict) else '',\n \"Verb\": verb,\n \"User\": username,\n \"SourceIPs\": source_ips,\n \"UserAgent\": eks_audit_log.get('userAgent', ''),\n \"ObjectRef\": object_ref_str,\n \"ResponseCode\": response_code,\n \"Stage\": stage,\n \"AuthDecision\": auth_decision,\n \"RawEvent\": simple_raw_event # Simple structure, no complex nested objects\n }\n \n return transformed_record\n \n except Exception as e:\n logger.error(f\"Error transforming EKS audit log: {str(e)}\")\n return None\n" } } }, "FirehoseLambdaInvokePermission": { "Type": "AWS::Lambda::Permission", "Properties": { "FunctionName": { "Ref": "EKSLogTransformFunction" }, "Action": "lambda:InvokeFunction", "Principal": "firehose.amazonaws.com", "SourceArn": { "Fn::GetAtt": [ "EKSAuditLogsDeliveryStream", "Arn" ] } } } }, "Outputs": { "SentinelRoleArn": { "Description": "Role ARN for Microsoft Sentinel AWS CloudWatch CloudTrail connector", "Value": { "Fn::GetAtt": [ "SentinelWebIdentityBasedRole", "Arn" ] } }, "SentinelSQSQueueURL": { "Description": "AWS SQS Queue URL for Microsoft Sentinel AWS CloudWatch CloudTrail connector", "Value": { "Ref": "SentinelSQSQueue" } }, "S3BucketName": { "Description": "S3 Bucket name where EKS logs will be stored", "Value": { "Ref": "S3Bucket" } }, "CloudWatchLogGroup": { "Description": "CloudWatch Log Group where EKS control plane logs will be stored", "Value": { "Fn::Sub": "/aws/eks/${EKSClusterName}/cluster" } }, "Step1EnableEKSAuditLogging": { "Description": "Step 1: Run this AWS CLI command to enable EKS audit logging (JSON format)", "Value": { "Fn::Sub": "aws eks update-cluster-config --region ${AWS::Region} --name ${EKSClusterName} --logging '{\"clusterLogging\":[{\"types\":[\"audit\"],\"enabled\":true}]}'" } }, "Step2CreateSubscriptionFilter": { "Description": "Step 2: Create CloudWatch Logs subscription filter to stream audit logs to Firehose", "Value": { "Fn::Sub": "aws logs put-subscription-filter --region ${AWS::Region} --log-group-name /aws/eks/${EKSClusterName}/cluster --filter-name EKSAuditLogsToFirehose --filter-pattern '' --destination-arn ${EKSAuditLogsDeliveryStream.Arn} --role-arn ${CloudWatchLogsRole.Arn}" } }, "FirehoseDeliveryStreamName": { "Description": "Kinesis Data Firehose delivery stream for EKS audit logs", "Value": { "Ref": "EKSAuditLogsDeliveryStream" } }, "CloudWatchLogsRoleArn": { "Description": "CloudWatch Logs role ARN for Firehose integration", "Value": { "Fn::GetAtt": [ "CloudWatchLogsRole", "Arn" ] } }, "SentinelSQSQueueArn": { "Description": "Log destination ARN to be used when setting up other accounts to export logs", "Value": { "Fn::GetAtt": [ "SentinelSQSQueue", "Arn" ] } }, "SentinelSQSQueueName": { "Description": "SQS Name", "Value": { "Fn::GetAtt": [ "SentinelSQSQueue", "QueueName" ] } }, "ProductionLambdaInfo": { "Description": "✅ Production Lambda function with SUCCESSFUL format", "Value": "This Lambda processes real EKS audit logs using the same format that works with Microsoft Sentinel ingestion (includes 'date' field and simplified RawEvent structure)." }, "NextSteps": { "Description": "Complete setup instructions for PRODUCTION use", "Value": "1) Run Step1 command to enable EKS audit logging, 2) Wait 5 minutes for logs to appear in CloudWatch, 3) Run Step2 command to create subscription filter, 4) Real EKS audit logs will be processed and appear in S3 success folders, 5) Logs should now be successfully ingested into Microsoft Sentinel AWSEKSLogs_CL table." } } }
2. Connect new collectors
To enable AWS Security Hub Connector for Microsoft Sentinel, click the Add new collector button, fill the required information in the context pane and click on Connect. - SentinelRoleArn: The AWS IAM Role ARN for cross-account access (e.g., arn:aws:iam::123456789012:role/SentinelRole) - SentinelSQSQueueURL: The full AWS EKS queue URL (e.g., https://sqs.region.amazonaws.com/account-id/queue-name)
3. Connect
Enable the AWS EKS connector. - Click 'Connect' to establish connection
Browse: 🏠 · Solutions · Connectors · Methods · Tables · Content · Parsers · ASIM Parsers · ASIM Products · 📊